{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [ANZ Chengdu Data Science Competition](http://www.dcjingsai.com/common/cmpt/ANZ%20Chengdu%20Data%20Science%20Competition_%E8%B5%9B%E4%BD%93%E4%B8%8E%E6%95%B0%E6%8D%AE.html?lang=en_US)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "You are provided with a dataset which is related to \n",
    "- direct marketing campaigns (phone calls) of a banking institution. \n",
    "- The business problem is to predict whether the client will subscribe to a term deposit. \n",
    "\n",
    "Participants are expected to \n",
    "- undertake a thorough analysis of the dataset and build a prediction model to solve this business problem.  \n",
    "- Business suggestions are welcomed but not mandatory.  \n",
    "- Participants will be required to submit analysis along with modelling/algorithm design and validation for judging.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:27.587058Z",
     "start_time": "2018-12-19T15:29:26.715091Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.20.1'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "sklearn.__version__\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:27.726823Z",
     "start_time": "2018-12-19T15:29:27.589203Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>job</th>\n",
       "      <th>marital</th>\n",
       "      <th>education</th>\n",
       "      <th>default</th>\n",
       "      <th>housing</th>\n",
       "      <th>loan</th>\n",
       "      <th>contact</th>\n",
       "      <th>month</th>\n",
       "      <th>day_of_week</th>\n",
       "      <th>...</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>poutcome</th>\n",
       "      <th>emp.var.rate</th>\n",
       "      <th>cons.price.idx</th>\n",
       "      <th>cons.conf.idx</th>\n",
       "      <th>euribor3m</th>\n",
       "      <th>nr.employed</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>32</td>\n",
       "      <td>management</td>\n",
       "      <td>married</td>\n",
       "      <td>professional.course</td>\n",
       "      <td>unknown</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>jul</td>\n",
       "      <td>mon</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.4</td>\n",
       "      <td>93.918</td>\n",
       "      <td>-42.7</td>\n",
       "      <td>4.962</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>41</td>\n",
       "      <td>admin.</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>cellular</td>\n",
       "      <td>apr</td>\n",
       "      <td>mon</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>93.075</td>\n",
       "      <td>-47.1</td>\n",
       "      <td>1.405</td>\n",
       "      <td>5099.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>32</td>\n",
       "      <td>admin.</td>\n",
       "      <td>married</td>\n",
       "      <td>university.degree</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>92.893</td>\n",
       "      <td>-46.2</td>\n",
       "      <td>1.299</td>\n",
       "      <td>5099.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37</td>\n",
       "      <td>admin.</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>unknown</td>\n",
       "      <td>unknown</td>\n",
       "      <td>cellular</td>\n",
       "      <td>jul</td>\n",
       "      <td>thu</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.4</td>\n",
       "      <td>93.918</td>\n",
       "      <td>-42.7</td>\n",
       "      <td>4.968</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>38</td>\n",
       "      <td>admin.</td>\n",
       "      <td>divorced</td>\n",
       "      <td>university.degree</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>jul</td>\n",
       "      <td>tue</td>\n",
       "      <td>...</td>\n",
       "      <td>8</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.4</td>\n",
       "      <td>93.918</td>\n",
       "      <td>-42.7</td>\n",
       "      <td>4.961</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   age         job   marital            education  default  housing     loan  \\\n",
       "0   32  management   married  professional.course  unknown       no       no   \n",
       "1   41      admin.   married          high.school       no      yes      yes   \n",
       "2   32      admin.   married    university.degree       no      yes       no   \n",
       "3   37      admin.   married          high.school       no  unknown  unknown   \n",
       "4   38      admin.  divorced    university.degree       no       no       no   \n",
       "\n",
       "    contact month day_of_week ...  campaign  pdays  previous     poutcome  \\\n",
       "0  cellular   jul         mon ...         4    999         0  nonexistent   \n",
       "1  cellular   apr         mon ...         1    999         0  nonexistent   \n",
       "2  cellular   may         mon ...         2    999         0  nonexistent   \n",
       "3  cellular   jul         thu ...         3    999         0  nonexistent   \n",
       "4  cellular   jul         tue ...         8    999         0  nonexistent   \n",
       "\n",
       "  emp.var.rate  cons.price.idx  cons.conf.idx  euribor3m  nr.employed   y  \n",
       "0          1.4          93.918          -42.7      4.962       5228.1  no  \n",
       "1         -1.8          93.075          -47.1      1.405       5099.1  no  \n",
       "2         -1.8          92.893          -46.2      1.299       5099.1  no  \n",
       "3          1.4          93.918          -42.7      4.968       5228.1  no  \n",
       "4          1.4          93.918          -42.7      4.961       5228.1  no  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('bank-additional-train.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:27.736542Z",
     "start_time": "2018-12-19T15:29:27.731142Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "37069"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:27.743117Z",
     "start_time": "2018-12-19T15:29:27.738573Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['age', 'job', 'marital', 'education', 'default', 'housing', 'loan',\n",
       "       'contact', 'month', 'day_of_week', 'duration', 'campaign', 'pdays',\n",
       "       'previous', 'poutcome', 'emp.var.rate', 'cons.price.idx',\n",
       "       'cons.conf.idx', 'euribor3m', 'nr.employed', 'y'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data\n",
    "\n",
    "a)Input variables:\n",
    "\n",
    "### bank client data:\n",
    "1. age (numeric)\n",
    "2. job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired','self-employed','services','student','technician','unemployed','unknown')\n",
    "3. marital : marital status (categorical: 'divorced','married','single','unknown'; note: 'divorced' means divorced or widowed)\n",
    "4. education (categorical: 'basic.4y','basic.6y','basic.9y','high.school','illiterate','professional.course','university.degree','unknown')\n",
    "5. default: has credit in default? (categorical: 'no','yes','unknown')\n",
    "6. housing: has home loan? (categorical: 'no','yes','unknown')\n",
    "7. loan: has personal loan? (categorical: 'no','yes','unknown')\n",
    "\n",
    "### related with the last contact of the current campaign:\n",
    "8. contact: contact communication type (categorical: 'cellular','telephone') \n",
    "9. month: last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec')\n",
    "10. day_of_week: last contact day of the week (categorical: 'mon','tue','wed','thu','fri')\n",
    "11. duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.\n",
    "\n",
    "### other attributes:\n",
    "12. campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)\n",
    "13. pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)\n",
    "14. previous: number of contacts performed before this campaign and for this client (numeric)\n",
    "15. poutcome: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success')\n",
    "\n",
    "### social and economic context attributes：\n",
    "16. emp.var.rate: employment variation rate. quarterly indicator (numeric)\n",
    "17. cons.price.idx: consumer price index. monthly indicator (numeric) \n",
    "18. cons.conf.idx: consumer confidence index. monthly indicator (numeric) \n",
    "19. euribor3m: euribor 3 month rate. daily indicator (numeric)\n",
    "20. nr.employed: number of employees. quarterly indicator (numeric)\n",
    "\n",
    "b)Output variable (desired target):\n",
    "\n",
    "21. y - has the client subscribed a term deposit? (binary: 'yes','no')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:27.750614Z",
     "start_time": "2018-12-19T15:29:27.745460Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    no\n",
       "1    no\n",
       "2    no\n",
       "3    no\n",
       "4    no\n",
       "Name: y, dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['y'][:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:27.837663Z",
     "start_time": "2018-12-19T15:29:27.752582Z"
    }
   },
   "outputs": [],
   "source": [
    "df[\"y\"][df[\"y\"] == \"no\"] = 0\n",
    "df[\"y\"][df[\"y\"] == \"yes\"] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:28.129288Z",
     "start_time": "2018-12-19T15:29:27.839641Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAETCAYAAAD6R0vDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAHUxJREFUeJzt3XuYXVWd5vHvS8JF0SYXYhpz6eCQBmnkZhlAuYVICOBj6G5FkJZARzMXHHHURwLddrjoM9Fx5NKtaaNEgi1EBB3SQAMxgLTOcEkEQkLERAgmEUgwJLQ3IPCbP9YqcixOpWpXrapdl/fzPOc5Z6+9zj6/OlXhZe+99tqKCMzMzErape4CzMxs4HG4mJlZcQ4XMzMrzuFiZmbFOVzMzKw4h4uZmRXncDEzs+IcLmZmVpzDxczMihtadwF12XvvvWPChAl1l2Fm1q8sX778uYgY1VG/QRsuEyZMYNmyZXWXYWbWr0h6qjP9fFjMzMyKc7iYmVlxDhczMyvO4WJmZsU5XMzMrDiHi5mZFedwMTOz4hwuZmZWnMPFzMyKG7RX6A9GE2bf2uX3rpt7asFKzGyg856LmZkV53AxM7PiHC5mZlacw8XMzIpzuJiZWXEOFzMzK87hYmZmxTlczMysOIeLmZkV53AxM7PiHC5mZlacw8XMzIpzuJiZWXG1h4ukYZJulPQzSaslHSVphKQlktbk5+G5ryRdJWmtpBWSDm/Yzozcf42kGfX9RGZmVnu4AFcCt0fEAcAhwGpgNrA0IiYCS/MywMnAxPyYBcwDkDQCmAMcAUwC5rQGkpmZ9b5a7+ciaS/gWOAcgIh4CXhJ0nTg+NxtIXAPcAEwHbg2IgK4L+/17JP7LomILXm7S4BpwPW99bMMdN25Fwz4fjBmg03dey77ApuBb0l6SNI3Je0JjI6Ip3OfZ4DR+fUYYH3D+zfktvbazcysBnWHy1DgcGBeRBwG/JYdh8AAyHspUeLDJM2StEzSss2bN5fYpJmZNVF3uGwANkTE/Xn5RlLYPJsPd5GfN+X1G4FxDe8fm9vaa/8jETE/IloiomXUqFFFfxAzM9uh1nCJiGeA9ZL2z01TgMeAxUDriK8ZwM359WLg7Dxq7EhgWz58dgcwVdLwfCJ/am4zM7Ma1HpCP/vvwHck7QY8AZxLCr0bJM0EngJOz31vA04B1gK/y32JiC2SLgMezP0ubT25b2Zmva/2cImIh4GWJqumNOkbwHntbGcBsKBsdWZm1hV1n3MxM7MByOFiZmbFOVzMzKw4h4uZmRXncDEzs+IcLmZmVpzDxczMinO4mJlZcQ4XMzMrzuFiZmbFOVzMzKw4h4uZmRXncDEzs+IcLmZmVpzDxczMinO4mJlZcQ4XMzMrzuFiZmbFOVzMzKw4h4uZmRXncDEzs+IcLmZmVpzDxczMiqs9XCStk/SopIclLcttIyQtkbQmPw/P7ZJ0laS1klZIOrxhOzNy/zWSZtT185iZWR8Il2xyRBwaES15eTawNCImAkvzMsDJwMT8mAXMgxRGwBzgCGASMKc1kMzMrPf1lXBpazqwML9eCJzW0H5tJPcBwyTtA5wELImILRHxPLAEmNbbRZuZWdIXwiWAOyUtlzQrt42OiKfz62eA0fn1GGB9w3s35Lb22s3MrAZDq3SWdBUwLyJWF6zh6IjYKOktwBJJP2tcGREhKUp8UA6vWQDjx48vsUkzM2ui6p7Lx4GVku6VdJak3bpbQERszM+bgB+Qzpk8mw93kZ835e4bgXENbx+b29prb/tZ8yOiJSJaRo0a1d3SzcysHVXD5YOkE+zvAa4FfiXpy5L278qHS9pT0ptbXwNTgZXAYqB1xNcM4Ob8ejFwdh41diSwLR8+uwOYKml4PpE/NbeZmVkNKoVLRNwUEVOB/YAvAS8BnwIek3SXpNMl7Vphk6OBH0t6BHgAuDUibgfmAidKWgO8Ny8D3AY8AawFvgH8t1zXFuAy4MH8uDS3mZlZDSqdc2kVEU8CF0r6HGkE138GpgDHAc9J+hbwjYj4RQfbeQI4pEn7r/P22rYHcF4721oALKj4o5iZWQ/o1mixiNjesDdzFPArYBTwWeBxSbdIemeBOs3MrB/p9lBkScdJug74EWn472bgCuDHwCnAfZI+1N3PMTOz/qNLh8XySfNzSMN6/xwQ8BPSFfPfi4iXc79JwPeBi4Hvdr9cMzPrD6pe53IMKVD+GtgD+A3wddK1L4+27R8RD+TzLxcUqNX6sQmzb+3ye9fNPbVgJWbWG6ruufwoP68i7aVcGxG/6eA9rdehmJnZIFH1nMsi4LiIeEdEfK0TwUJE/HNE7Nu18szMrD+qtOcSER/uqULMzGzgqLTnImmUpGNbr6pvsv5P8vq9y5RnZmb9UdXDYn8P/CvwSjvrX8nrL+xOUWZm1r9VDZcTSfdN+V2zlRHxW+BO0v1VzMxskKoaLuOAnU7pQpr7a1wHfczMbACrGi4BdDTN/m7AkK6VY2ZmA0HVcHmcnRzykqS8fm13ijIzs/6tarjcCBwg6Z8kvaFxRV7+J2B/PNWLmdmgVvUK/auAM4H/Cpwm6V7S1fdjgGOBtwKPkCauNDOzQarqRZS/l3Q88DXgdOCMhtWvAtcBH4+I3xer0MzM+p3KsyJHxFbgw5LOB94FDAO2Ag9ExHOF6zMzs36oS1PuA0TEZtJth83MzP5It28WZmZm1lblPRdJI4C/BSYBw2l+TUtExJRu1mZmZv1U1ZuFHQDcA4wi3X2yPdGNmszMrJ+reljsy8BbgC8CbwN2jYhdmjx8hb6Z2SBWNVyOAW6NiIsiYl1EtDc7ciWShkh6SNIteXlfSfdLWivpu5J2y+275+W1ef2Ehm1cmNsfl+SJM83MalQ1XAQ81gN1nA+sblj+InB5ROwHPA/MzO0zgedz++W5H5IOJF1z8xfANOBrkrz3ZGZWk6rhspw0vUsxksYCpwLfzMsCTiBNNQOwEDgtv56el8nrp+T+04FFEfFiRDxJmttsUsk6zcys86qGy6XAKfkq/VKuAD5LusIfYCSwNSK25+UNpOllyM/rAfL6bbn/a+1N3mNmZr2s6lDkccDNwJ2SriftyWxt1jEiru1oY5LeB2yKiOWFA6u9z5sFzAIYP358T3+cmdmgVTVcriENMxbwkfxoO+xYua3DcAHeA7xf0inAHsCfAFcCwyQNzXsnY0mTY5KfxwEbJA0F9gJ+3dDeqvE9r4mI+cB8gJaWFg+XNjPrIVXD5dySHx4RFwIXAuQ9l89ExFmSvgd8AFgEzCDtLQEszsv/L6+/KyJC0mLgOklfIc3MPBF4oGStZmbWeVVnRV7Yca8iLgAWSfo88BBwdW6/Gvi2pLXAFvKszBGxStINpJFs24HzSg2TtvpNmH1rl9+7bu6pBSsxs87q8sSVpUXEPaSr/4mIJ2gy2isi/gB8sJ33fwH4Qs9VaGZmndWlcJE0Cvhr4O3AnhHx0Yb2fYFHfU8XM7PBqysTV84k3ZFyD3acvP9oXj2adD5kFjsOZZmZ2SBT6ToXSSeSRlv9HPhLYF7j+ohYCaxix0WPZmY2CFXdc7kAeBo4LiJekHRYkz4rgKO6XZmZmfVbVa/QbwFuiYgXdtJnA/CnXS/JzMz6u6rhshvw2w76DAM8DNjMbBCrGi7rgHd20OcI4PEuVWNmZgNC1XC5GThGUtNrTSSdCxwM3NTdwszMrP+qekL/S6Sr4q+X9AHS3F5I+jjpRmJ/BawB/rFkkbZDd65WNzPrLVWnf3le0nGkSSkb916uys//Dnw4Ijo6L2NmZgNY5YsoI+KXwPGSDiYNOR5Juq/KfRGxvHB9ZmbWD3V5brGIWEG6psXMzOyPVD2hb2Zm1qFKey6S/qGTXSMiLutCPWZFebp+s3pUPSx28U7Wtd7ZsXUyS4eLmdkgVTVcJrfTPgx4F/AJ4Fbgn7tTlJmZ9W9VhyL/aCerb5b0XdLthRd1qyozM+vXip7Qj4hHSVfxX1Ryu2Zm1r/0xGixXwIH9cB2zcysn+iJcDkC8C2OzcwGsapDkcfvZDvjgI8BRwM3dLMuMzPrx6qOFlvHjiHHzYg0ceVnulqQmZn1f1XD5Vqah8urwPOkkWI3R8SL3S3MzMz6r6pDkc8p+eGS9gDuBXbPtdwYEXMk7UsazjwSWA58JCJekrQ7KeDeCfwa+FBErMvbuhCYSboL5ici4o6StZqZWefVPbfYi8AJEXEIcCgwTdKRwBeByyNiP9Ie0czcfybwfG6/PPdD0oGk+8z8BTAN+JqkIb36k5iZ2WtqDZdIfpMXd82PAE4AbsztC4HT8uvpeZm8fook5fZFEfFiRDwJrAUm9cKPYGZmTVQdLXZXFz8nImJKO9scQjr0tR/wVeAXwNaI2J67bADG5NdjgPV5g9slbSMdOhsD3New2cb3NH7WLGAWwPjx7Q18MzOz7qp6Qv/4/BykkWFt7ay9qYh4BThU0jDgB8ABFWvqtIiYD8wHaGlp2dmoNzMz64aqh8X2ABYDTwLnAvsCb8jPfws8QZr+ZfeI2KXh0eH5j4jYCtxNurvlMEmtwTcW2JhfbyRdT0NevxfpxP5r7U3eY2ZmvaxquHwOaAFaImJhRDyVz3M8FRHXkK7On5T7dUjSqLzHgqQ3ACcCq0kh84HcbQYpsCAF24z8+gPAXRERuf0MSbvnkWYTScOizcysBlUPi50F3JT3Ml4nIrZIuhH4G6AzNxbbB1iYz7vsAtwQEbdIegxYJOnzwEPA1bn/1cC3Ja0FtpBGiBERqyTdADwGbAfOy4fbzMysBlXD5a3ASx30eZkUGh2KiBXAYU3an6DJaK+I+APwwXa29QXgC535XLPO6M5dLMF3srTBrephsQ3AdEm7NVuZL3Kcjs93mJkNalXDZSFpyPBdko5tvVBR0hBJxwFLgbcB1xSt0szM+pWqh8XmkqZeeT/ppPurkrYAI0hBJdLJ9bklizQzs/6l0p5LRLwcEaeRTtjfBWwjBcs20l7LWRFxWsMFkGZmNghV3XMBICKuA64rXIuZmQ0QdU9caWZmA1CX9lwkHQx8GHg7sGdEvDe3TyANIV4SEc8XqtHMzPqZyuEi6VLgInbs9TTO0bULcD3wSeAfu12dmZn1S5UOi0k6A/h7YAnp/iv/s3F9vvhxGWk0mZmZDVJVz7l8gnSvlOn56vpmV+uvJs3tZWZmg1TVcHkHcEdE7GwKmF8Bo7tekpmZ9XdVw0XAqx30GQ38oWvlmJnZQFA1XNYA725vpaRdgKOBVd0pyszM+reqo8VuAD4v6dMR8b+brL+INPfYld2uzKyf686syp5R2fq7quFyBWnK+y9JOp08DFnSl4FjSDcSu498K2EzMxucKoVLRPxe0mTSnslZQOvtiz9FOhfzL8DHPbeYmdngVvkiyojYBpwj6VPAu4CRpIkrH4iIzYXrMzOzfqhSuEg6G3g2Iu6IiC3AHT1TlpmZ9WdVR4stAKb1RCFmZjZwVA2XZ7rwHjMzG2SqBsXtwOR8PYuZmVlTVUPi74A3A1dL2rsH6jEzswGgarhcTxoZdjawXtJqSXdLuqvNY2lnNiZpXH7/Y5JWSTo/t4+QtETSmvw8PLdL0lWS1kpaIenwhm3NyP3XSJpR8ecyM7OCqg5FPr7h9e7A/vnRVjRpa2Y78OmI+KmkNwPLJS0BzgGWRsRcSbOB2cAFwMmkGZcnAkcA84AjJI0A5pAu4oy8ncW+YZmZWT12uuci6ROSJrUuR8QunXwM2dl2G7b3dET8NL/+D9J0/WOA6cDC3G0hcFp+PR24NpL7gGGS9gFOIt39cksOlCV4VJuZWW06Oix2BQ3/kZb0iqTP9UQh+RbJhwH3A6Mj4um86hl2TOE/Bljf8LYNua29djMzq0FH4fIH0uGvVsqPoiS9CbgJ+GREvNC4LiKCzh9m6+hzZklaJmnZ5s2eTMDMrKd0FC5PAidJarz5V5H/0LeStCspWL4TEd/Pzc/mw13k5025fSMwruHtY3Nbe+1/JCLmR0RLRLSMGjWq5I9hZmYNOgqXrwOHA7+S9EpuuzgfHtvZo1MTV0oScDWwOiK+0rBqMdA64msGcHND+9l51NiRwLZ8+OwOYKqk4Xlk2VQ8NY2ZWW12OlosIq6StAk4FXgrMBn4JbCu0Oe/B/gI8Kikh3PbRcBc4AZJM4GngNPzutuAU4C1wO+Ac3OdWyRdBjyY+12a5z4zM7MadDgUOSIWAYsAJL0KfCsiLi3x4RHxY9o/hzOlSf8AzmtnWwtIc5+ZmVnNql5EeQlwTw/UYWZmA0jVm4Vd0lOFmJnZwOEJKM3MrDiHi5mZFedwMTOz4hwuZmZWnMPFzMyKc7iYmVlxDhczMyvO4WJmZsU5XMzMrDiHi5mZFedwMTOz4hwuZmZWnMPFzMyKc7iYmVlxDhczMyuu0v1crIwJs2+tuwQzsx7lPRczMyvOey5mfVB39m7XzT21YCVmXeM9FzMzK87hYmZmxTlczMysuFrDRdICSZskrWxoGyFpiaQ1+Xl4bpekqyStlbRC0uEN75mR+6+RNKOOn8XMzHaoe8/lGmBam7bZwNKImAgszcsAJwMT82MWMA9SGAFzgCOAScCc1kAyM7N61BouEXEvsKVN83RgYX69EDitof3aSO4DhknaBzgJWBIRWyLieWAJrw8sMzPrRXXvuTQzOiKezq+fAUbn12OA9Q39NuS29tpfR9IsScskLdu8eXPZqs3M7DV9MVxeExEBRMHtzY+IlohoGTVqVKnNmplZG30xXJ7Nh7vIz5ty+0ZgXEO/sbmtvXYzM6tJXwyXxUDriK8ZwM0N7WfnUWNHAtvy4bM7gKmShucT+VNzm5mZ1aTW6V8kXQ8cD+wtaQNp1Ndc4AZJM4GngNNz99uAU4C1wO+AcwEiYouky4AHc79LI6LtIAEzM+tFtYZLRJzZzqopTfoGcF4721kALChYmpmZdUNfPCxmZmb9nMPFzMyKc7iYmVlxDhczMyvO4WJmZsU5XMzMrDiHi5mZFedwMTOz4hwuZmZWnMPFzMyKc7iYmVlxDhczMyvO4WJmZsU5XMzMrDiHi5mZFedwMTOz4hwuZmZWnMPFzMyKc7iYmVlxQ+suoL+aMPvWukswa6o7f5vr5p5asBIbzLznYmZmxXnPxcyK6O7evPeaBpYBFS6SpgFXAkOAb0bE3JpLMutXfLjXShkw4SJpCPBV4ERgA/CgpMUR8Vi9lZlZZ9QVbN5j6hkDJlyAScDaiHgCQNIiYDrgcDGzdnkARM8YSOEyBljfsLwBOKKmWsxsEHAwtW8ghUuHJM0CZuXF30h6vJc+em/guV76rM7qizWB66qiL9YEfbOuPleTvgj0wbrouKY/68xGBlK4bATGNSyPzW2viYj5wPzeLApA0rKIaOntz92ZvlgTuK4q+mJN0Dfr6os1Qd+sq1RNA+k6lweBiZL2lbQbcAawuOaazMwGpQGz5xIR2yV9HLiDNBR5QUSsqrksM7NBacCEC0BE3AbcVncdTfT6obhO6Is1geuqoi/WBH2zrr5YE/TNuorUpIgosR0zM7PXDKRzLmZm1kc4XAqSNE7S3ZIek7RK0vm5fYSkJZLW5OfhvVzXHpIekPRIruuS3L6vpPslrZX03TwQoldJGiLpIUm39KGa1kl6VNLDkpbltlp/h7mGYZJulPQzSaslHVVnXZL2z99R6+MFSZ/sI9/V/8h/6yslXZ//DdT6tyXp/FzPKkmfzG29/l1JWiBpk6SVDW1N61ByVf7OVkg6vLOf43Apazvw6Yg4EDgSOE/SgcBsYGlETASW5uXe9CJwQkQcAhwKTJN0JPBF4PKI2A94HpjZy3UBnA+sbljuCzUBTI6IQxuGZNb9O4Q0b97tEXEAcAjpe6utroh4PH9HhwLvBH4H/KDOmgAkjQE+AbRExEGkAT5nUOPflqSDgI+RZhI5BHifpP2o57u6BpjWpq29Ok4GJubHLGBepz8lIvzooQdwM2mus8eBfXLbPsDjNdb0RuCnpNkLngOG5vajgDt6uZax+Q/5BOAWQHXXlD93HbB3m7Zaf4fAXsCT5POkfaWuhjqmAj/pCzWxY7aOEaRBS7cAJ9X5twV8ELi6YflzwGfr+q6ACcDKjv6OgK8DZzbr19HDey49RNIE4DDgfmB0RDydVz0DjK6hniGSHgY2AUuAXwBbI2J77rKB9I+yN11B+gf2al4e2QdqAgjgTknL86wOUP/vcF9gM/CtfBjxm5L27AN1tToDuD6/rrWmiNgIfBn4JfA0sA1YTr1/WyuBYySNlPRG4BTSRd995ffXXh3NptXq1PfmcOkBkt4E3AR8MiJeaFwXKf57fYheRLwS6fDFWNKu+QG9XUMjSe8DNkXE8jrraMfREXE46ZDAeZKObVxZ0+9wKHA4MC8iDgN+S5tDKHX9beVzF+8Hvtd2XR015fMF00mB/FZgT15/GKhXRcRq0mG5O4HbgYeBV9r0qeX311apOhwuhUnalRQs34mI7+fmZyXtk9fvQ9p7qEVEbAXuJh0WGCap9Vqn102X08PeA7xf0jpgEenQ2JU11wS89n++RMQm0jmESdT/O9wAbIiI+/PyjaSwqbsuSCH804h4Ni/XXdN7gScjYnNEvAx8n/T3VuvfVkRcHRHvjIhjSed8fk7931Wr9urocFqt9jhcCpIk4GpgdUR8pWHVYmBGfj2DdC6mN+saJWlYfv0G0nmg1aSQ+UAddUXEhRExNiImkA6p3BURZ9VZE4CkPSW9ufU16VzCSmr+HUbEM8B6Sfvnpimk20nUWld2JjsOiUH9Nf0SOFLSG/O/ydbvqu6/rbfk5/HAXwHXUf931aq9OhYDZ+dRY0cC2xoOn+1cb53QGgwP4GjS7uQK0m7vw6RjqyNJJ67XAD8ERvRyXQcDD+W6VgL/kNvfBjwArCUd0ti9pu/teOCWvlBT/vxH8mMV8He5vdbfYa7hUGBZ/j3+H2B43XWRDjn9Gtiroa0vfFeXAD/Lf+/fBnbvA39b/04KuUeAKXV9V6T/EXgaeJm0RzyzvTpIg2y+SjpH+yhpBF6nPsdX6JuZWXE+LGZmZsU5XMzMrDiHi5mZFedwMTOz4hwuZmZWnMPFzMyKc7iYmVlxDhczMyvO4WJmZsU5XMwKk3SOpJskPSHp9/kOjT+R9Dft9H+XpDsl/Ufu+8N8l8mLJYWk45u85wBJ10haL+klSc9Kuq5h7jGzWg3tuIuZVTSPNC/ZvaQ5nEaS5pj7tqT9I+JzrR3zdP53ku6W+H3SHE7vIE2yeFezjUualvvuCvwraa6ssaTJEE+VNDkiftozP5pZ53huMbPCJP2niPhFm7bdgH8DjgUmRMRGSbuQ7uy3H3BKRPxbQ///wo5byk6OiHty+3DgCdK9QI6NiMca3nMQcB/w80j3ozGrjQ+LmRXWNlhy20uk2WWHkqaAB3g3KVjubgyWbD7pfh9tnQ0MA+Y0Bkv+jJXAN4DDJB3YrR/CrJt8WMyssHy/jgtIITIeeEObLq23iT0sP/+47TYi4lVJ/xf48zarjsrPh0i6uMnHt/Z/O2l6d7NaOFzMCpLUes+Q4aT7d9xJuof7K8AE0o2Yds/d98rPz9Jcs/aR+fljHZTyps5VbNYzHC5mZX2KFADnRsQ1jSskncmOu/0BvJCfR7ezrWbt2/LzIRGxoht1mvUon3MxK2u//HxTk3XHtVl+KD8f3bZjPtn/7ibbuC8/H9Ol6sx6icPFrKx1+fn4xkZJJwEfbdP3J6Shx5Mlndxm3Sxef74F4FvAVmCOpEltV0rapdl1MWa9zUORzQqSdDDwIBDAjcCvgIOAacANwIeASyLi4tz/eOB20nUuN5HC5mDgRNJ1LicDx0XEvQ2fMQX4Aem8ylLSNTUBjCOd8B8ZEXv07E9qtnM+52JWUESskDQZ+DxwKunf2COkCxy3ksKlsf89ko5r6A9wPzAZOCsvv9DmPUtziH0GOIl0iOwlUpDdRfNDcma9ynsuZn2UpJ8ARwB7RcRv667HrAqfczGrkaQ3ShrWpP0c0gn9Ox0s1h95z8WsRpIOII0aW0KaI2wo6eLKo0mH0d4dEavrq9CsaxwuZjXKc4X9L9Iw5T8lXWD5DPBD4AvNppIx6w8cLmZmVpzPuZiZWXEOFzMzK87hYmZmxTlczMysOIeLmZkV53AxM7Pi/j89/7EvsVs16QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(df.age, bins = 20)\n",
    "plt.xlabel('age', fontsize = 20)\n",
    "plt.ylabel('frequency', fontsize = 20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# One-hot encoding\n",
    "\n",
    "https://scikit-learn.org/stable/modules/preprocessing.html#encoding-categorical-features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:28.185327Z",
     "start_time": "2018-12-19T15:29:28.131355Z"
    }
   },
   "outputs": [],
   "source": [
    "dat = pd.get_dummies(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:28.192930Z",
     "start_time": "2018-12-19T15:29:28.187503Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['age', 'duration', 'campaign', 'pdays', 'previous', 'emp.var.rate',\n",
       "       'cons.price.idx', 'cons.conf.idx', 'euribor3m', 'nr.employed',\n",
       "       'job_admin.', 'job_blue-collar', 'job_entrepreneur', 'job_housemaid',\n",
       "       'job_management', 'job_retired', 'job_self-employed', 'job_services',\n",
       "       'job_student', 'job_technician', 'job_unemployed', 'job_unknown',\n",
       "       'marital_divorced', 'marital_married', 'marital_single',\n",
       "       'marital_unknown', 'education_basic.4y', 'education_basic.6y',\n",
       "       'education_basic.9y', 'education_high.school', 'education_illiterate',\n",
       "       'education_professional.course', 'education_university.degree',\n",
       "       'education_unknown', 'default_no', 'default_unknown', 'default_yes',\n",
       "       'housing_no', 'housing_unknown', 'housing_yes', 'loan_no',\n",
       "       'loan_unknown', 'loan_yes', 'contact_cellular', 'contact_telephone',\n",
       "       'month_apr', 'month_aug', 'month_dec', 'month_jul', 'month_jun',\n",
       "       'month_mar', 'month_may', 'month_nov', 'month_oct', 'month_sep',\n",
       "       'day_of_week_fri', 'day_of_week_mon', 'day_of_week_thu',\n",
       "       'day_of_week_tue', 'day_of_week_wed', 'poutcome_failure',\n",
       "       'poutcome_nonexistent', 'poutcome_success', 'y_0', 'y_1'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dat.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:28.223403Z",
     "start_time": "2018-12-19T15:29:28.195583Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>duration</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>emp.var.rate</th>\n",
       "      <th>cons.price.idx</th>\n",
       "      <th>cons.conf.idx</th>\n",
       "      <th>euribor3m</th>\n",
       "      <th>nr.employed</th>\n",
       "      <th>...</th>\n",
       "      <th>day_of_week_fri</th>\n",
       "      <th>day_of_week_mon</th>\n",
       "      <th>day_of_week_thu</th>\n",
       "      <th>day_of_week_tue</th>\n",
       "      <th>day_of_week_wed</th>\n",
       "      <th>poutcome_failure</th>\n",
       "      <th>poutcome_nonexistent</th>\n",
       "      <th>poutcome_success</th>\n",
       "      <th>y_0</th>\n",
       "      <th>y_1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>32</td>\n",
       "      <td>565</td>\n",
       "      <td>4</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>93.918</td>\n",
       "      <td>-42.7</td>\n",
       "      <td>4.962</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>41</td>\n",
       "      <td>982</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>93.075</td>\n",
       "      <td>-47.1</td>\n",
       "      <td>1.405</td>\n",
       "      <td>5099.1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>32</td>\n",
       "      <td>188</td>\n",
       "      <td>2</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>92.893</td>\n",
       "      <td>-46.2</td>\n",
       "      <td>1.299</td>\n",
       "      <td>5099.1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37</td>\n",
       "      <td>56</td>\n",
       "      <td>3</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>93.918</td>\n",
       "      <td>-42.7</td>\n",
       "      <td>4.968</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>38</td>\n",
       "      <td>419</td>\n",
       "      <td>8</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>93.918</td>\n",
       "      <td>-42.7</td>\n",
       "      <td>4.961</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 65 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  duration  campaign  pdays  previous  emp.var.rate  cons.price.idx  \\\n",
       "0   32       565         4    999         0           1.4          93.918   \n",
       "1   41       982         1    999         0          -1.8          93.075   \n",
       "2   32       188         2    999         0          -1.8          92.893   \n",
       "3   37        56         3    999         0           1.4          93.918   \n",
       "4   38       419         8    999         0           1.4          93.918   \n",
       "\n",
       "   cons.conf.idx  euribor3m  nr.employed ...   day_of_week_fri  \\\n",
       "0          -42.7      4.962       5228.1 ...               0.0   \n",
       "1          -47.1      1.405       5099.1 ...               0.0   \n",
       "2          -46.2      1.299       5099.1 ...               0.0   \n",
       "3          -42.7      4.968       5228.1 ...               0.0   \n",
       "4          -42.7      4.961       5228.1 ...               0.0   \n",
       "\n",
       "   day_of_week_mon  day_of_week_thu  day_of_week_tue  day_of_week_wed  \\\n",
       "0              1.0              0.0              0.0              0.0   \n",
       "1              1.0              0.0              0.0              0.0   \n",
       "2              1.0              0.0              0.0              0.0   \n",
       "3              0.0              1.0              0.0              0.0   \n",
       "4              0.0              0.0              1.0              0.0   \n",
       "\n",
       "   poutcome_failure  poutcome_nonexistent  poutcome_success  y_0  y_1  \n",
       "0               0.0                   1.0               0.0  1.0  0.0  \n",
       "1               0.0                   1.0               0.0  1.0  0.0  \n",
       "2               0.0                   1.0               0.0  1.0  0.0  \n",
       "3               0.0                   1.0               0.0  1.0  0.0  \n",
       "4               0.0                   1.0               0.0  1.0  0.0  \n",
       "\n",
       "[5 rows x 65 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dat.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:28.255274Z",
     "start_time": "2018-12-19T15:29:28.226532Z"
    }
   },
   "outputs": [],
   "source": [
    "X = dat.drop(['y_0', 'y_1'], axis = 1)\n",
    "y = dat['y_1']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hold-out Samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:28.316901Z",
     "start_time": "2018-12-19T15:29:28.257808Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, random_state=1, train_size = 0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:28.353806Z",
     "start_time": "2018-12-19T15:29:28.319619Z"
    }
   },
   "outputs": [],
   "source": [
    "?train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:28.361512Z",
     "start_time": "2018-12-19T15:29:28.356196Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((29655, 63), (29655,), (37069, 63))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtrain.shape, ytrain.shape, X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-17T08:53:24.261509Z",
     "start_time": "2018-12-17T08:53:24.249083Z"
    }
   },
   "source": [
    "### Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:28.431720Z",
     "start_time": "2018-12-19T15:29:28.363778Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB # BernoulliNB\n",
    "model = GaussianNB()\n",
    "model.fit(Xtrain, ytrain)\n",
    "y_model = model.predict(Xtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:41:50.994865Z",
     "start_time": "2018-12-19T15:41:50.988635Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.86147828432694906"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score, roc_auc_score,  roc_curve, auc\n",
    "accuracy_score(ytest, y_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T16:53:44.635924Z",
     "start_time": "2018-12-19T16:53:44.429034Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xd4VGX2wPHvSYDQQQhFKYKA9KJGLKBYABEQ1wqI66LuCggWWBVQFCy4qGBBSsC6VnRRlB9VUBBFRUJTqtIJ0nuHJOf3x71JZkIyGUIm087nefJw73vbmcvMnHnfe+/7iqpijDHG5CQm2AEYY4wJbZYojDHG+GSJwhhjjE+WKIwxxvhkicIYY4xPliiMMcb4ZIkiAohINxH5JthxBJuIVBeRwyISW4DHrCEiKiKFCuqYgSQiK0TkmjxsF7HvQRG5RkSSgx1HMFmiyGcislFEjrlfWNtF5H0RKRnIY6rqx6raNpDHCEXuuW6dPq+qm1W1pKqmBjOuYHETVu2z2YeqNlTVubkc57TkGK3vwWhhiSIwblLVkkAz4CJgYJDjyZNg/kqOlF/oZ8LOtwlVligCSFW3AzNxEgYAIhInIsNFZLOI7BCRRBEp5rH8ZhFZKiIHRWSdiLRzy8uIyDsisk1EtorIC+lNLCLSXUR+dKfHishwzzhE5GsR6edOnyciX4jILhHZICIPe6w3REQmishHInIQ6J71NblxfOBuv0lEBolIjEcc80VklIgcEJHVInJ9lm19vYb5IvKaiOwBhohILRH5TkT2iMhuEflYRMq6638IVAf+z629PZH1l66IzBWR5939HhKRb0Qk3iOee9zXsEdEns5aQ8nyuouJyAh3/QMi8qPn/xvQzf0/3S0iT3ls11xEfhaR/e7rHiUiRTyWq4j0FpE/gT/dsjdEZIv7HlgkIld5rB8rIk+6741D7vJqIjLPXWWZez46u+t3dN9P+0XkJxFp4rGvjSLSX0R+A46ISCHPc+DGnuTGsUNEXnU3TT/WfvdYV3i+B91tG4rILBHZ6277ZA7nNcfPgxvbAo//z17iNI0Vdef/J06t/YCIzBORhh77fV9ExojIdDfG+SJSWUReF5F97nvzoiznYqCIrHSXv5d+nGxizvEzFLFU1f7y8Q/YCLR2p6sCvwNveCx/DZgMlANKAf8H/Mdd1hw4ALTBSeJVgHrusknAOKAEUBH4FejhLusO/OhOXw1sAcSdPwc4Bpzn7nMR8AxQBLgAWA/c4K47BDgF/M1dt1g2r+8D4Gs39hrAH8D9HnGkAH2BwkBn9/WU8/M1pAAPAYWAYkBt91zEARVwvqBez+5cu/M1AAUKufNzgXXAhe7+5gLD3GUNgMNAS/dcDHdfe+sc/l9Hu9tXAWKBK9240o/5lnuMpsAJoL673SXA5e5rqgGsAh712K8Cs3DeD8XcsruB8u42/wa2A0XdZY/jvKfqAuIer7zHvmp77PsiYCdwmRvzP9xzFudx/pYC1TyOnXFOgZ+Bv7vTJYHLszvP2bwHSwHb3NiLuvOX5XBefX0eYtz/8yFAHWAfcJHHtve528QBrwNLPZa9D+x2z39R4DtgA3CPey5eAOZkeS8td89FOWA+8IK77Bog2SOmHD9DkfoX9AAi7c99wx0GDrkfpm+Bsu4yAY4AtTzWvwLY4E6PA17LZp+VcL58inmUdU1/o2f5kAqwGbjanf8X8J07fRmwOcu+BwLvudNDgHk+XlsscBJo4FHWA5jrEcdfuEnKLfsV+Lufr2FzTsd21/kbsCTLuc4tUQzyWP4gMMOdfgb41GNZcfe1nZYo3C+HY0DTbJalH7NqltfcJYfX8CgwyWNegetyed370o8NrAFuzmG9rIliLPB8lnXWAK08zt992bx/0xPFPOBZID6H15xToujq+f/k43X5/Dx4HGsvToId6GNfZd2Yyrjz7wNveSx/CFjlMd8Y2J/ldff0mG8PrHOnryEzUfj8DEXqn7VLBsbfVHW2iLQCPgHigf04v4qLA4tEJH1dwfkCBufXzLRs9nc+zi/0bR7bxeDUHLyoqorIBJwP6zzgLuAjj/2cJyL7PTaJBX7wmD9tnx7i3Tg2eZRtwvmVnW6rup8ej+Xn+fkavI4tIpWAN4CrcH45xuB8aZ6J7R7TR3F+GePGlHE8VT0qTpNXduJxfpWuO9PjiMiFwKtAAs7/fSGcX6Sesr7ux4D73RgVKO3GAM57xFccns4H/iEiD3mUFXH3m+2xs7gfeA5YLSIbgGdVdYofx/U3xtw+D6jqRhGZg/PFPTpjJafJcihwh7ufNHdRPE4tFmCHx7GOZTOf9SYTz3OR/r7Nyp/PUMSxaxQBpKrf4/yySb9msBvnDdpQVcu6f2XUufANzhu1Vja72oLzazzeY7vSqtowm3UBPgVuF5HzcX4BfeGxnw0e+yirqqVUtb1n2D5e0m6c5pnzPcqqA1s95quIx6feXf6Xn68h67FfdMsaq2ppnCYZ8bH+mdiG0zQIONcgcJp7srMbOE72/ze5GQusBuq4r+FJvF8DeLwO93rEE8CdwDmqWhbniy99m5zeI9nZAgzN8v9dXFU/ze7YWanqn6raFaeZ8CVgooiU8LWNx3Ev8CO+3D4PiEgHnFrGt8ArHtveBdwMtAbK4NQ84PRzeyaqeUynv2+z8uczFHEsUQTe60AbEWmqqmk4bdmviUhFABGpIiI3uOu+A9wrIteLSIy7rJ6qbgO+AUaISGl3WS23xnIaVV2C8yF8G5ipqum/fn4FDrkXCYu5F0Ybicil/rwQdW47/RwYKiKl3ETUj8waCzhfKg+LSGERuQOoD0w709fgKoXTjHdARKrgtM972oF/X0jZmQjcJCJXinNxeQg5fMm4/2/vAq+6FzJj3Qu4cX4cpxRwEDgsIvWAXn6snwLsAgqJyDM4NYp0bwPPi0gdcTQRkfQEl/V8vAX0FJHL3HVLiEgHESnlR9yIyN0iUsF9/envoTQ3tjRyPvdTgHNF5FH3YnUpEbks60q5fR7EufHgbeCfONdXbhKR9C/kUjg/PPbg1Epe9Oc15aK3iFQVkXLAU8Bn2axzVp+hcGWJIsBUdRfOBeBn3KL+wFrgF3HuLJqNc2ESVf0VuBfnAt8B4Hsyf73fg9NssBKn+WUicK6PQ3+C82vrE49YUoGOOHdhbSAzmZQ5g5f0EE678nrgR3f/73osX4Bz4XE3TtPA7aqa3qRzpq/hWeBinHMxFfgyy/L/AIPEuaPnsTN4DajqCve1TMCpXRzGufB7IodNHsO5iLwQp838Jfz7/DyG8+v3EM6XYnZfPp5mAjNwbhLYhFOT8WwSeRUnWX+Dk4DewbmIDk6y+697Pu5U1SSca1SjcM73WrK5k82HdsAKETmM0wTYRVWPqepRnP/b+e6xLvfcSFUP4dyEcBNOk9yfwLU5HCPHzwMwHvhaVae576H7gbfdxPiBe3624ryffjmD15WTT3DO63qcprMXsq6QT5+hsJN+Z4wxZ01EugP/VNWWwY7lTInzUOR+nCaiDcGOxxQsEdmI896dHexYQpHVKEzUEpGbRKS42+4+HKfGsDG4URkTeixRmGh2M84Fy79wmsu6qFWxjTmNNT0ZY4zxyWoUxhhjfAq7B+7i4+O1Ro0awQ7DGGPCyqJFi3araoW8bBt2iaJGjRokJSUFOwxjjAkrIrIp97WyZ01PxhhjfLJEYYwxxidLFMYYY3yyRGGMMcYnSxTGGGN8skRhjDHGp4DdHisi7+L0srhTVRtls1xweqRsjzPQS3dVXRyoeIwxJuqMrQxHnfGaLqnKJXndTSBrFO/jdFOckxtx+tepAzyAM8CLMcaY/HJ0B/nRS1PAahSqOk9EavhY5WbgA7cTtl9EpKyInOsOcGOMMSYvjmyHP79k/5JJPPlFB9JUSLzdnxFscxbMJ7Or4D0gS7JbdlqiEJEHcGodVK9evUCCM8aYsOEmB/74H7r5ez5b2pC+k29g+yFnMMN7Epbhe3h038KiCw9VHY8z2hUJCQnW3a0xxngkB7Z8Dyhrd5fjwS/vZtYf3sOqf76sIeGaKLbiPZh5VbfMGGNMdrJJDgAnUmJ56buWvPjdVZxIyfxaP6/0Qd64eQa3NVnJpW/k/bDBTBSTgT4iMgG4DDhg1yeMMSaLHJJDuu/+rEmvLzvwx674jLKYGOGhh5rzXPV/UVqTzzqEQN4e+ylwDRAvIsnAYKAwgKomAtNwbo1di3N77L2BisUYY8JKLskB4Nipwjww9X4++rGyV3lCwnkkJnbgkkvOw7O5adFjsiiv4QTyrqeuuSxXoHegjm+MMWHFj+QAAlWvggvvpGjtW9j6zSzSh3kvXTqOF1+8jp49E4iNzd8nH8LiYrYxxkSkM0wO1LkVSp6bXsrYsR1o0iSRW26px6uv3sB555UKSJiWKIwxpiDlITkcpjyjR/9K34YVKeKxVt268axe3ZuaNc8JaMiWKIwxJtDOoubw9dereeihCWzZcpC0NGXgwKu8tgp0kgBLFMYYExhnkRwANm8+wMMPT+frr9dklD333Dz+/vemVK1aOvDxe7BEYYwx+eUskwPAqVOpjBy5gMGD53LkyKmM8vj44owY0ZYqVQJzHcIXSxTGGHM28iE5pPvll2R69JjCb7/t8Cr/5z8v4qWX2lCuXLH8j98PliiMMeZM5WNyANi37xgDB37L+PGLvHp7bdSoIomJHWjRIrh93FmiMMYYf/idHK6GC+/INTl4euutxYwbl/k8XLFihRgy5Br69r2cwoVj8yf+s2CJwhhjchLA5ODpkUcu4913l7BmzR46dryQN9+8kRo1yp51+PnFEoUxxngKcHI4fjyFAweOU6lSyYyyuLhCvP12J3btOsLf/lYPZwDQ0GGJwhhjCqjmMGvWOh58cBo1a5Zl5sy7vRJCy5ahO9aOJQpjTHQqoOQAsH37Yfr1m8mnny4HYO3avUyYsJyuXRvnPf4CZInCGBM90pPDms8heR6BTA4AaWnKuHFJDBz4LQcOnMgoL1MmjtTU8BmDzRKFMSayFXBySLd06XZ69pzCggXe47HddVdjRoxoS+XKJXPYMvRYojDGRJ4gJQeAQ4dOMHjwXN54YwFpaZnHrV27HGPGtKdNm1o+tg5NliiMMZEhiMkh3YEDx2nceCxbthzMKCtSJJaBA1syYEBLihYNz6/c8IzaGGMgJJKDpzJlitK69QW8995SAK67riZjxrSnbt34XLYMbZYojDHhJcSSQ1Yvv9yGBQu28uSTLbnrrsYh90xEXliiMMaEvhBMDvPnb+aZZ+YyceIdnHNOZmd98fHF+f33XsTEhH+CSGeJwhgTmkIwOQDs2XOUAQNm8/bbSwAYMGA248bd5LVOJCUJsERhjAklIZocAFSVDz5YxmOPzWL37qMZ5Z98spznn7+OihVLFEgcwWCJwhgTXGeaHC68DUpULtAQV63aRa9eU/n++01e5TffXJeRI2+M6CQBliiMMcEQBskB4NixUwwd+gMvvzyfU6fSMsqrVSvNm2/eyM031yvwmILBEoUxpmCESXJI9+2363nggSmsX78voyw2Vujb93IGD76GkiWLBC22gmaJwhgTOGGWHDz99dchryRxxRVVSUzsSJMmlYIYVXBYojDG5K8wTg6e7r67Ce++u5Rly7bz0kutuf/+iyPubiZ/WaIwxpy9ME8OixdvIy1NSUg4L6NMRHjvvZspXrxwxF+szo0lCmNM3oR5cgA4ePAEzzwzhzff/JX69eNZvLgHRYpkjlEdSsORBpMlCmOM/yIgOYDzTMQXX6zikUdm8NdfhwBYsWIXr732M/37twxydKHHEoUxxrcISQ7p1q/fR58+05g+fa1XeevWF3DrrfWDFFVos0RhjDldhCUHgJMnUxkx4ieee24ex4+nZJRXqlSC1167gS5dGkVEB36BYInCGOOIwOSQbt68TfTsOYVVq3ZnlIlAr14JDB16PWXLFg1idKEvoIlCRNoBbwCxwNuqOizL8urAf4Gy7joDVHVaIGMyxng4sh3++AL++F/EJYd0u3Yd4YYbPvKqRTRrVpnExA5cdlnVIEYWPgKWKEQkFhgNtAGSgYUiMllVV3qsNgj4XFXHikgDYBpQI1AxGWM4s+RQ906n470wSw6eKlQowYABLRgy5HtKlizC889fS58+zSlUKCbYoYWNQNYomgNrVXU9gIhMAG4GPBOFAqXd6TLAXwGMx5joFUXJYf/+46c1JfXv35Ldu4/Sv39LqlYtncOWJieBTBRVgC0e88nAZVnWGQJ8IyIPASWA1tntSEQeAB4AqF69er4HakxEiqLkAHD06CleeGEeo0cvZPHiB6hVq1zGsqJFC/Hmm+2DGF14C/bF7K7A+6o6QkSuAD4UkUaqmua5kqqOB8YDJCQkZPduN8ZA1CWHdNOn/0nv3tPYsGE/AL17T2P69G52F1M+CWSi2ApU85iv6pZ5uh9oB6CqP4tIUSAe2BnAuIyJLFGaHAC2bj3Io4/OZOLElV7lhw+f5ODBE5QpY3cz5YdAJoqFQB0RqYmTILoAd2VZZzNwPfC+iNQHigK7AhiTMZEhipMDQGpqGqNHL2TQoO84dOhkRnm5csV4+eXW3HvvRVHbgV8gBCxRqGqKiPQBZuLc+vquqq4QkeeAJFWdDPwbeEtE+uK807urqjUtGZOdKE8O6ZKS/qJHjyksXrzNq/wf/2jKK6+0oUKF6O7ALxACeo3CfSZiWpayZzymVwItAhmDMWHNkoOXV16ZT//+s/H8OVmvXjxjx3bgmmtqBC2uSBfsi9nGmKwsOeSoefMqGUmiaNFCDBp0FY8/3sKrx1eT/yxRGBMKLDn4pVWrGnTv3oxt2w4xenR7r1tgTeBYojAmWCw55OjEiRSGD/+J2rXL0blzI69lY8d2IC4u1m59LUCWKIwpSJYccvX99xvp2XMqq1fvpmLFErRtW4tzzimWsbxoUfvaKmh2xo0JNH+TQ7VWTsd7UZgcwOm87/HHZ/Hf/y7LKNu584h7G+zVQYzMWKIwJhAsOfgtLU15990lPPHELPbtO55RXqpUEV544Tp69740iNEZsERhTP6x5HDGli/fSc+eU5g/f4tX+R13NOC1126gShXrwC8U+JUoRKQIUF1V1+a6sjHRxJJDnhw9eopnn53Lq6/+QkpKZtduNWuWZdSo9rRvXyeI0Zmsck0UItIBeBUoAtQUkWbAYFW9JdDBGROSLDmctWPHTvHuu0szkkThwjE8/viVPPXU1RQvXjjI0Zms/KlRPIfTPfgcAFVdKiK1AxqVMaHGkkO+Kl++OMOHt6F796+56qrqJCZ2pEGDCsEOy+TAn0RxSlX3Z7ln2fpjMpHPkkO+SElJ47vvNtC2bS2v8nvuaUq5csXo2PFCeyYixPmTKFaJyJ1AjNsT7MPAL4ENy5ggseSQrxYsSKZHjyksW7aDH364l5YtMwceExFuuqluEKMz/vInUfQBngHSgC9xeoN9MpBBGVOgLDnku/37j/Pkk9+SmJiU0TdTz55TWLKkB4ULW79M4cafRHGDqvYH+qcXiMitOEnDmPBkySEgVJVPP11Ov34z2bHjSEZ5sWKFuPvuJkGMzJwNfxLFIE5PCk9lU2ZMaLPkEFB//rmHBx+cxuzZ673K27evw6hRN1Kz5jlBisycrRwThYjcgDNMaRURedVjUWmcZihjQp8lh4A7cSKFYcN+5D//+ZETJ1Izys87rxQjR7bj1lvr28XqMOerRrETWA4cB1Z4lB8CBgQyKGPOiiWHAtW79zTeeWdJxnxMjPDww8157rlrKVUqLoiRmfySY6JQ1SXAEhH5WFWP57SeMSHBn+QgMU6vrJYc8tUTT7Tgww9/4+TJVC699DzGjevIRRedG+ywTD7y5xpFFREZCjQAiqYXquqFAYvKGH9YcihwaWlKamqa151LF15YnqFDr6NYsUL07JlAbGxMECM0geBPongfeAEYDtwI3Is9cGeCxZJD0Cxbtp2ePady000X8uSTV3kte+yxK4MUlSkI/iSK4qo6U0SGq+o6YJCIJAFPBzg2YxyWHILq8OGTDBkyl9df/4XUVGXp0u106dKICy6wu5iihT+J4oSIxADrRKQnsBUoFdiwTNSz5BASvv56NQ89NJ0tWw5mlKWmpvHzz1ssUUQRfxJFX6AETtcdQ4EywH2BDMpEKUsOIWPTpv08/PAMJk9e41V+zTU1GDu2A/XqxQcpMhMMuSYKVV3gTh4C/g4gIlUCGZSJIpYcQsqpU6m88cYCBg+ey9GjpzLK4+OLM2JEW/7+9yb2TEQU8pkoRORSoArwo6ruFpGGOF15XAdULYD4TCSy5BCSNm3aT6dOE/jttx1e5f/618UMG9aacuWKBSkyE2y+nsz+D3AbsAznAvYU4EHgJaBnwYRnIoYlh5BXuXJJTp7MfLK6UaOKJCZ2oEWL6j62MtHAV43iZqCpqh4TkXLAFqCxqq73sY0xmSw5hJW4uEIkJnagQ4dPGDy4FY8+ern19GoA34niuKoeA1DVvSLyhyUJkytLDmFhzZrd/Pe/yxg69Dqvaw6tWtVg8+a+1sxkvPhKFBeISHoPsYIzXnZGj7GqemtAIzPhw5JD2Dh+PIX//OcHhg2bz8mTqTRuXJGuXRt7rWNJwmTlK1HclmV+VCADMWHmjJLDnW5yqFTgYZpMs2at48EHp7F27d6Msscfn8VttzWgSBFrYjI589Up4LcFGYgJA5YcwtL27Yfp128mn3663Ku8efMqjBvX0ZKEyZU/D9yZaGbJIWylpqYxfvwiBg78lgMHTmSUlykTx7BhrfnXvy62DvyMXwKaKESkHfAGEAu8rarDslnnTmAIzjfQMlW9K5AxGT9kJIfPIfkHLDmEnyVLttGz51R+/XWrV/lddzVmxIi2VK5cMkiRmXDkd6IQkThVPZH7mhnrxwKjgTZAMrBQRCar6kqPdeoAA4EWqrpPRCr6H7rJV5YcIsqYMQu9kkSdOuUYM6YDrVtfEMSoTLjKNVGISHPgHZw+nqqLSFPgn6r6UC6bNgfWpt9SKyITcJ7NWOmxzr+A0aq6D0BVd575SzB5ZskhYg0b1ppJk1Zz6NBJnnyyJf37t6RoUWtpNnnjzztnJNAR+ApAVZeJyLV+bFcF5yG9dMnAZVnWuRBARObjNE8NUdUZfuzb5JUlh4izadN+SpWK87qttXz54nz88a3UrHkOF15YPojRmUjgT6KIUdVNWToCS81p5Twcvw5wDU7fUfNEpLGq7vdcSUQeAB4AqF7duhM4Y5YcItKpU6m89tovPPvs93Tr1pjx42/yWn7DDbWDFJmJNP4kii1u85O61x0eAv7wY7utQDWP+apumadkYIGqngI2iMgfOIljoedKqjoeGA+QkJBgo+v5w5JDRJs/fzM9e05l+XKntfattxbTvXszrryyWi5bGnPm/EkUvXCan6oDO4DZblluFgJ1RKQmToLoAmS9o+kroCvwnojE4zRFWTcheWXJIeLt2XOUAQNm8/bbS7zKmzSpRFycPQ9hAsOfRJGiql3OdMeqmiIifYCZONcf3lXVFSLyHJCkqpPdZW1FZCVOc9bjqrrnTI8V1Sw5RAVV5YMPlvHYY7PYvftoRnmJEoV59tlreOSRyylUyJ6JMIEhqr5bckRkHbAG+Az4UlUPFURgOUlISNCkpKRghhB8lhyiyqpVu+jVayrff7/Jq/xvf6vHG2+0o3r1MkGKzIQTEVmkqgl52dafEe5qiciVOE1Hz4rIUmCCqk7IywFNHllyiErLlm3n0kvf4tSptIyyatVKM2pUezp1qhvEyEw08evGalX9CfhJRIYArwMfA5YoAs3v5NDKo1dWSw6RpEmTSrRsWZ05czYSGyv063cFzzzTipIliwQ7NBNF/HngriTOg3JdgPrA18CVAY4rellyiGqq6jU+hIgwdmwHevacyhtvtKNJE/u/NgXPnxrFcuD/gJdV9YcAxxOdLDlEvdTUNMaOTeKTT35n7tzuXj261q0bz5w5/whidCba+ZMoLlDVtNxXM2fEkoNxLVr0Fz16TGHRom0AjBjxEwMHXhXkqIzJlGOiEJERqvpv4AsROe1bzEa4ywNLDsbDwYMnePrp7xg1aiFpaZnvhf/9byVPPNHCugA3IcNXjeIz918b2e5sWHIwWagqEyeu5JFHZrBt2+GM8ri4WJ566ipLEibk+Brh7ld3sr6qeiUL90E6GwEvJ5YcTA7Wr99Hnz7TmD59rVd5mzYXMGZMB2rXLhekyIzJmT/XKO7j9FrF/dmURTdLDiYXr776M0899R3Hj6dklFWqVILXX29H584NydLxpjEhw9c1is44t8TWFJEvPRaVAvZnv1WUseRgzsDhwyczkoQI9OqVwNCh11O2bNEgR2aMb75qFL8Ce3B6fR3tUX4IWJLtFpFqbGU4usP/9S05mGz079+Cjz/+neLFCzNuXEeaN68S7JCM8YuvaxQbgA04vcVGN3+ShCUH40pLU95/fylXX32+1zWHuLhCfPPN3VSpUto68DNhxVfT0/eq2kpE9uHdpiKAqqpddQOodq0lB5NhxYqd9Oo1lR9+2EzbtrWYMaOb17WH888vG8TojMkbX01P6cOdxhdEIGHrzu+CHYEJAUePnuL5579n+PCfSUlxnk/95pt1fPnlKm67rUGQozPm7Phqekp/Grsa8JeqnhSRlkAT4CPgYAHEZ0zImzbtT3r3nsbGjZn3eBQqFMNjj13BjTfWCWJkxuQPf26P/Qq4VERqAe8BU4BPgI6BDMyYULd160EefXQmEyeu9Cpv2bI6iYkdaNiwYpAiMyZ/+ZMo0lT1lIjcCrypqiNFJHrueko5nvOy4nZNIhqlpKQxevSvDBo0h8OHT2aUlytXjFdeaUP37s2IibFnIkzk8GsoVBG5A/g78De3rHDgQgoxyd9nTpetDff94dwEb6LWokV/8eijM73KundvxiuvtCE+vniQojImcPy5R+8+nAvbL6vqehGpCXwa2LBCyLr/y5yudZMlCcNll1Xl3nubAVC/fjxz5/6D99672ZKEiVj+DIW6XEQeBmqLSD1graoODXxoIUAV1k/JnL/ALstEG1Vly5aDp41L/fJkWPxQAAAfw0lEQVTLbahXL55HH73ca+wIYyJRrjUKEbkKWAu8A7wL/CEiLQIdWEjYswIOugPaFykNVVoGNx5ToNau3Uu7dh+TkDCevXuPeS2Ljy/OE0+0sCRhooI/TU+vAe1VtYWqXgl0AN4IbFghwrPZqUY7iLVxiqPBiRMpvPDCPBo1GsM336xj166jDBxoHRSY6OXPxewiqppx/5+qrhKR6PjG9Gx2qmXNTtFgzpwN9Oo1lTVr9mSUxcQIRYsWOm08a2OihT+JYrGIJOI8ZAfQjWjoFPDobvjrZ2daYqDGjcGNxwTUzp1HePzxWXzwwTKv8ksuOZfExI4kJJwXpMiMCT5/EkVP4GHgCXf+B+DNgEUUKjZMI6OLq3OvgOLWk0kkSktT3nlnMf37z2bfvsxnZkqVKsKLL15Pr14JNtqciXo+E4WINAZqAZNU9eWCCSlE2N1OUeH22z9n0qTVXmV33tmQ1167gfPOKxWkqIwJLTn+VBKRJ3G67+gGzBKR+wosqmBLPQkbPR6oqnVT8GIxAXXLLfUypi+44BymT+/GZ5/dbknCGA++ahTdgCaqekREKgDTcG6PjXzJP8BJt8/D0jWgvPX+GanuvrsJH3/8OwkJ5/HUU1dRrFj0dDpgjL98JYoTqnoEQFV3iUj0NNRmbXayO13CXnLyQR55ZAZ9+15Oy5bVM8pFhGnTulnfTMb44CtRXOAxVrYAtTzHzlbVWwMaWbCowvos3XaYsJWSksabby7gmWfmcvjwSdas2c3ixT28HpSzJGGMb74SxW1Z5kcFMpCQsXcN7F/nTBcu4QxvasLSggXJ9OgxhWXLMoeyXbFiF7NmraNDhwuDGJkx4cXXwEXfFmQgIcOz2en8tlAoLnixmDzZt+8YTz75LePGLUI9BvFt0KACiYkduOqq84MXnDFhyJ/nKKKLNTuFLVXl00+X07fvTHbuPJJRXqxYIZ55phX9+l1hfTMZkwcBvUAtIu1EZI2IrBWRAT7Wu01EVEQSAhlPro7tha3zM+drtg9eLOaMrF+/jzZtPqRbty+9kkT79nVYseJBBgxoaUnCmDzyu0YhInGqeuIM1o8FRgNtgGRgoYhM9uw3yl2vFPAIsMDffQfMxpmgqc505eZQwkawCxepqWn8+OPmjPkqVUoxcuSN3HJLPeufyZiz5E83481F5HfgT3e+qYj404VHc5yxK9ar6klgAnBzNus9D7wE+BhztIBYs1PYqlOnPAMHtiQmRujb93JWrerNrbfWtyRhTD7wp+lpJNAR2AOgqstwRrzLTRVgi8d8sluWQUQuBqqp6lRfOxKRB0QkSUSSdu3a5ceh8yAtBTZMz5y3bjtC1o4dh5k8ec1p5QMGtGTx4gd49dUbKFXKbkIwJr/4kyhiVHVTlrLUsz2w+wDfq8C/c1tXVceraoKqJlSoUOFsD529v36CE/ud6ZJVoULTwBzH5FlamjJuXBL16o2mc+eJrFu312t5XFwhmjatHKTojIlc/iSKLSLSHFARiRWRR4E//NhuK1DNY76qW5auFNAImCsiG4HLgclBu6DtNTa2PY0dapYt206LFu/Ss+dU9u8/zvHjKfTuPQ31vP/VGBMQ/lzM7oXT/FQd2AHMdstysxCoIyI1cRJEF+Cu9IWqegDI6LtbROYCj6lqkr/B5yvrLTYkHT58kiFD5vL667+QmpqZFGrVOod+/a6waxDGFIBcE4Wq7sT5kj8jqpoiIn2AmUAs8K6qrhCR54AkVZ18xtEGyr61sNftarpQMah2XXDjMQB89dVqHnpoOsnJBzPKCheOYcCAlgwc2NI68DOmgOSaKETkLTJG8Mmkqg/ktq2qTsPpddaz7Jkc1r0mt/0FjGdtonprKFwsaKEY2LRpPw8/POO0C9bXXFODsWM7UK+eDSJlTEHyp+nJc1T5osAteN/NFP5sbOyQoarcccf/WLjwr4yy+PjivPpqW+6+u4k1NRkTBP40PX3mOS8iHwI/BiyignbiACR/nzlfs0PwYjGICMOHt6VVq/cB+Ne/LmbYsNaUK2e1PGOCJS99PdUEIueR5Y3fOM9QAFS8GEpV8b2+yVcHDhyndOk4r5rC1Vefz3PPXcN119WkRYvqOW9sjCkQ/jyZvU9E9rp/+4FZwMDAh1ZA7G6noFBVPvroN+rUeZNPPvn9tOVPP93KkoQxIcJnjUKcn3lNyXz+IU0j6cb1tFTY4HGt3brtKBBr1uymV6+pzJmzEYB+/b6hffs6nHOONS8ZE4p81ijcpDBNVVPdv8hJEgDbFsCx3c50icpQ6eLgxhPhjh9PYfDgOTRpkpiRJACKFIll/fp9wQvMGOOTP9colorIRaq6JODRFDTPZqeaHSCKhgUvaLNmrePBB6exdm1mtxuxscIjj1zGs89eS8mSRYIYnTHGlxwThYgUUtUU4CKcLsLXAUdwxs9WVQ3/n9/WW2zAbd9+mH79ZvLpp8u9yi+7rAqJiR1p1sz6ZjIm1PmqUfwKXAx0KqBYCtaBjbDb/fKKjYPq1wc1nEg0Z84GbrnlMw4cyBzGpEyZOIYNa80DD1xCTIw9E2FMOPCVKARAVdcVUCwFa71Hz+bVroUiJYMXS4Rq0qQShQtnjirXrVtjRoxoS6VKdq6NCSe+EkUFEemX00JVfTUA8RQca3YKuPLlizN8eBuGDv2BsWM7cP31FwQ7JGNMHvhKFLFASdyaRUQ5eRi2zMmcv8Cexj4bqsqkSatZvHgbL7zg3aHiPfc0pUuXRsTF5eXZTmNMKPD16d2mqs8VWCQFadNsSD3pTMc3htLnBzeeMLZx43769JnG1Kl/AtCuXW1atsx8UE5ELEkYE+Z83Q8aeTWJdNbsdNZOnUrlpZd+pEGD0RlJAmDw4LnBC8oYExC+fupF5m1AmuZ9Idu67ThjP/64mZ49p7BiReb45SLQo8clvPhiZL5tjIlmOSYKVd2b07Kwtj0Jju5wpotVgMrNgxtPGNmz5yj9+8/mnXe8n71s2rQSiYkdufzyqkGKzBgTSNHXeOzVCWB7iInNeV0DOBerP/hgGY89Novdu49mlJcoUZjnnruWhx++jEKF7Kl2YyJVlCcKa3by14cf/uaVJP72t3qMHNmOatXKBDEqY0xBiK6fgYeSYafbbBJTGM5vG9x4woSIMGZMB4oUiaV69TJ8/XUXJk3qbEnCmCgRXTUKz4vYVVtBXOngxRLCvvtuAy1bVqdIkcxmuQsvLM+UKV258spqlChhHfgZE02iq0ZhY2P79Ndfh+jceSLXX/8Bw4f/dNryNm1qWZIwJgpFT6I4dRQ2z86ct+sTGVJT03jzzQXUqzeKzz9fAcDzz8+zMSKMMUA0NT1t/g5SjjvT5epD2VrBjSdELFr0Fz16TGHRom1e5Xfe2ZBSpaz2YIyJpkRhdzt5OXDgOE8/PYfRoxeSlpY5cGHduuUZO7YD115bM4jRGWNCSXQkCtUs1yeit9sOVWXixJU88sgMtm07nFEeFxfLoEFX8/jjV1rfTMYYL9HxjbBzKRze6kwXPQfOuyK48QTRJ5/8zt13T/Iqa9u2FqNHt6d27XJBisoYE8qi42K2Z22ixo0QEx35MTu3396AunXLA1C5ckkmTLiNGTO6WZIwxuQoOr4xo7i32JMnU72eh4iLK8TYsR344otVDB16HWXKFA1idMaYcBD5ieLIdti+0JmWWKhxQ3DjKSC7dh3h8cdnsW3bYWbM6IZIZq/x115b0y5WG2P8FvmJYv20zOkqLZ1rFBEsLU15770lPPHEbPbuPQbAZ5+toEuXRkGOzBgTriL/GkUUNTutWLGTVq3e55///L+MJAEwb96mIEZljAl3kV2jSDkOm2Zlzkfo8xNHj57i+ee/Z/jwn0lJScsor1GjLKNG3UiHDhcGMTpjTLgLaI1CRNqJyBoRWSsiA7JZ3k9EVorIbyLyrYjk7+DVW+bCqSPO9Dl1oFzdfN19KJg69Q8aNhzDsGHzM5JEoUIxDBjQghUrHrQkYYw5awGrUYhILDAaaAMkAwtFZLKqrvRYbQmQoKpHRaQX8DLQOd+CiOCnsVNT0+jceSJffLHKq7xly+okJnagYcOKQYrMGBNpAlmjaA6sVdX1qnoSmADc7LmCqs5R1fTRcH4B8m8szaxPY0dYooiNjaFs2cxbW8uVK8Y773Ti+++7W5IwxuSrQCaKKsAWj/lktywn9wPTs1sgIg+ISJKIJO3atcu/o+9eDgfdi7hFSkOVq/zbLoy89FJr4uOL0717M9as6cN9911ETIzkvqExxpyBkLjrSUTuBhKAV7JbrqrjVTVBVRMqVKjg3069nsZuB7GFzz7QIDlw4DhPPDHL604mgPLli7N6dW/ee+9m4uOLByk6Y0ykC+RdT1uBah7zVd0yLyLSGngKaKWqJ/Lt6BEwSJGq8vnnK3j00Zls336Y/fuPM3689y2+5ctbgjDGBFYgaxQLgToiUlNEigBdgMmeK4jIRcA4oJOq7sy3Ix/dBX/97B4kxunfKcysXbuXdu0+pkuXL9i+3enl9a23FvP77zuCHJkxJtoErEahqiki0geYCcQC76rqChF5DkhS1ck4TU0lgf+5XUxsVtVOZ33wDdMBd4yFc6+A4vFnvcuCcuJECi+/PJ+hQ3/gxInUjPJzzy3JG2+0o1Eju1BtjClYAX3gTlWnAdOylD3jMd06IAcO07ud5szZQK9eU1mzZk9GWUyM0Lv3pbzwwnWULh0XxOiMMdEq8p7MTj0JG2dkzodBtx07dx7hsce+4cMPf/Mqv+SSc0lM7EhCwnlBiswYYyIxUST/ACcPOdOla0D5BkENxx+zZq3zShKlShXhxRevp1evBGJjQ+LGNGNMFIu8b6GszU4S+s8V3HVXY669tgYAd9zRgNWr+9CnT3NLEsaYkBBZNQrVkO8t9siRk/z11yHq1CmfUSYijB3bgfXr93HjjXWCGJ0xxpwushLF3jWwf50zXbgkVG0V3Hiy+L//W0OfPtMpWbIIS5b08Bp5rm7deOrWDZ+7s4wx0SOy2jY8axM12kKh0LhLaMuWA9xyy2d06jSBzZsPsHLlLkaM+CnYYRljjF8iq0YRYrfFpqSkMXLkAp55Zg5HjpzKKI+PL0716mWCGJkxxvgvchLFsb2wdb47I1CzfVDDWbAgmR49prBsmfeT1PfffxEvvdTaut4wxoSNyEkUG2eAuk8yn9scSlQKShj79h3jySe/Zdy4RahmljdsWIHExI60bFk9KHEZY0xeRU6iCIFmpxMnUrjoonFs2nQgo6xYsUIMHtyKvn2v8Lp4bYwx4SIyLmanpbj9O7mClCji4gpx330XZcx36FCHlSt7079/S0sSxpiwFRk1iq3z4cR+Z7pkVajQtEAOq6pIlgf6+vdvwbx5m3jwwUu55ZZ6py03xphwExk1iqxjTxTAl/Ps2etp1mwca9fu9SqPiyvE7Nn3cOut9S1JGGMiQmTUKArw+sSOHYfp1+8bPvnkdwB6957GjBndLCkY48OpU6dITk7m+PHjwQ4l4hUtWpSqVatSuHD+jeoZ/oli31rYu9qZLlQMql0XkMOkpSnjxy9iwIDZHDiQORDfggXJrF+/j1q1ygXkuMZEguTkZEqVKkWNGjXsR1UAqSp79uwhOTmZmjVr5tt+w7/pybM2Ub01FC6W74dYtmw7LVq8S69eU72SRNeujVi9uo8lCWNycfz4ccqXL29JIsBEhPLly+d7zS38axQBHBv78OGTDBkyl9df/4XU1MyHImrXLseYMe1p06ZWvh7PmEhmSaJgBOI8h3eiOHEAkr/PnK/ZId92PXv2eu6992uSkw9mlBUpEsuAAS0YOPAqihYN71NnjDH+Cu9vu43fOM9QAFS8GEpVybddFykS65Ukrr22BmPGdKBePevh1ZiAGlsZju44vbx4Jei1Pc+7jY2NpXHjxqSkpFCzZk0+/PBDypYtC8CKFSt46KGH2Lp1K2lpadxzzz0MGjQo49f59OnTefrppzl69ChxcXFcd911jBgxIs+xhJvwvkbh2VtsPt/tdPXV53Pvvc2oUKE4H354C99+e48lCWMKQnZJwle5n4oVK8bSpUtZvnw55cqVY/To0QAcO3aMTp06MWDAANasWcOyZcv46aefGDNmDADLly+nT58+fPTRR6xcuZKkpCRq1659VrFklZKSkq/7y2/hW6NIS4X10zLnz2KQop9+2sK+fcfo0OFCr/Lhw9syfHhbypXL/wvkxkStEWfRhp7btv9W38tdV1xxBb/95gw//Mknn9CiRQvatm0LQPHixRk1ahTXXHMNvXv35uWXX+app56iXr16gFMz6dWr12n7PHz4MA899BBJSUmICIMHD+a2226jZMmSHD58GICJEycyZcoU3n//fbp3707RokVZsmQJLVq04Msvv2Tp0qUZtZw6derw448/EhMTQ8+ePdm8eTMAr7/+Oi1atPDrdeaX8E0U2xbA8T3OdInKUOniM97F3r3HGDBgNm+9tZgKFYqzenUfr6RgCcKYyJOamsq3337L/fffDzjNTpdcconXOrVq1eLw4cMcPHiQ5cuX8+9//zvX/T7//POUKVOG3393nrHat29frtskJyfz008/ERsbS2pqKpMmTeLee+9lwYIFnH/++VSqVIm77rqLvn370rJlSzZv3swNN9zAqlWr8vDK8y58m56yNjuJ/y9FVfnww2XUqzeKt95aDMCuXUd5+unv8jtKY0yIOHbsGM2aNaNy5crs2LGDNm3a5Ov+Z8+eTe/evTPmzznnnFy3ueOOO4iNdfqB69y5M5999hkAEyZMoHPnzhn77dOnD82aNaNTp04cPHgwo4ZSUMK3RpHHp7HXrNlNr15TmTNno1d5p0516d+/ZT4FZ4zJUW7NQ76al/xsWspO+jWKo0ePcsMNNzB69GgefvhhGjRowLx587zWXb9+PSVLlqR06dI0bNiQRYsW0bRp3vqQ87xdNevzDSVKlMiYvuKKK1i7di27du3iq6++YtCgQQCkpaXxyy+/ULRo0TwdPz+EZ43iwEbYvdyZjo2D81vnusmxY6d45pk5NGmS6JUkqlUrzVdfdebrr7vYqHPGhILiOYwlk1P5me6+eHFGjhzJiBEjSElJoVu3bvz444/Mnj0bcGoeDz/8ME888QQAjz/+OC+++CJ//PEH4HxxJyYmnrbfNm3aZFwgh8ymp0qVKrFq1SrS0tKYNGlSjnGJCLfccgv9+vWjfv36lC9fHoC2bdvy5ptvZqy3dOnSszwDZy48E4XX09jXQeESOa8LfPPNOho3Hsvzz8/j5ElncKPYWOHf/76ClSt7c/PN9QIZrTHmTPTa7tQcsv6dxa2xWV100UU0adKETz/9lGLFivH111/zwgsvULduXRo3bsyll15Knz59AGjSpAmvv/46Xbt2pX79+jRq1Ij169efts9Bgwaxb98+GjVqRNOmTZkzZw4Aw4YNo2PHjlx55ZWce+65PuPq3LkzH330UUazE8DIkSNJSkqiSZMmNGjQINskFWiimveqXDAkJCRo0sB42DjTKbh+NDR7MMf1jxw5Sc2ab7Br19GMsssvr0piYgeaNq0c6HCNMcCqVauoX79+sMOIGtmdbxFZpKoJedlf+NUoNA22zMmcv8D309glShRhxAjntreyZYuSmNiB+fPvsyRhjDF+Cr+L2ScPQupJZ7pCEyh9vtfiLVsOUK2a97WGu+9uwpYtB7n//ouoVKlkQUVqjDERIfxqFOkj2YHX3U6HDp2gb98Z1Kz5Bj/8sMlrExHhySevsiRhTBCFWzN3uArEeQ7DRHEgc/qCjqgqX3yxkvr1R/P66wtITVV69ZqacdHaGBN8RYsWZc+ePZYsAix9PIr8vpU2/Jqe0jsBLFaBDcfq0Kfjp0yb9qfXKpUrl2T//uNUrOj7bihjTMGoWrUqycnJ7Nq1K9ihRLz0Ee7yU/glCuBkSiyvJnXlucfGcexYZmdaFSuW4LXXbqBr10bW970xIaRw4cL5OuKaKVgBvT1WRNoBbwCxwNuqOizL8jjgA+ASYA/QWVU3+tpnvYpxWijmn6zYUdFjP9CjxyW8+OL1nHOO9c9kjDFZnc3tsQGrUYhILDAaaAMkAwtFZLKqrvRY7X5gn6rWFpEuwEtA59P3lmnNrvJAZpJo2rQSiYkdufzy/K1qGWOMcQTyYnZzYK2qrlfVk8AE4OYs69wM/NednghcL7m0GaUvLFHkJK92mkFS0gOWJIwxJoACeY2iCrDFYz4ZuCyndVQ1RUQOAOWB3Z4ricgDwAMA5YtDjXJDAPh4MfQrHLsoEMGHiXiynKsoZucik52LTHYuMtXN64ZhcTFbVccD4wFEJGn3kby1s0UaEUnKa5tjpLFzkcnORSY7F5lEJCmv2way6WkrUM1jvqpblu06IlIIKINzUdsYY0yICGSiWAjUEZGaIlIE6AJMzrLOZOAf7vTtwHdqT+QYY0xICVjTk3vNoQ8wE+f22HdVdYWIPAckqepk4B3gQxFZC+zFSSa5GR+omMOQnYtMdi4y2bnIZOciU57PRdh1M26MMaZghV9fT8YYYwqUJQpjjDE+hWyiEJF2IrJGRNaKyIBslseJyGfu8gUiUqPgoywYfpyLfiKyUkR+E5FvReT87PYTCXI7Fx7r3SYiKiIRe2ukP+dCRO503xsrROSTgo6xoPjxGakuInNEZIn7OWkfjDgDTUTeFZGdIrI8h+UiIiPd8/SbiFzs145VNeT+cC5+rwMuAIoAy4AGWdZ5EEh0p7sAnwU77iCei2uB4u50r2g+F+56pYB5wC9AQrDjDuL7og6wBDjHna8Y7LiDeC7GA73c6QbAxmDHHaBzcTVwMbA8h+Xtgek4nVxcDizwZ7+hWqMISPcfYSrXc6Gqc1Q1fVDwX3CeWYlE/rwvAJ7H6TfseEEGV8D8ORf/Akar6j4AVd1ZwDEWFH/OhQKl3ekywF8FGF+BUdV5OHeQ5uRm4AN1/AKUFZFzc9tvqCaK7Lr/qJLTOqqaAqR3/xFp/DkXnu7H+cUQiXI9F25VupqqTi3IwILAn/fFhcCFIjJfRH5xe3OORP6ciyHA3SKSDEwDHiqY0ELOmX6fAGHShYfxj4jcDSQArYIdSzCISAzwKtA9yKGEikI4zU/X4NQy54lIY1Xd73OryNQVeF9VR4jIFTjPbzVS1bRgBxYOQrVGYd1/ZPLnXCAirYGngE6qeqKAYitouZ2LUkAjYK6IbMRpg50coRe0/XlfJAOTVfWUqm4A/sBJHJHGn3NxP/A5gKr+DBTF6TAw2vj1fZJVqCYK6/4jU67nQkQuAsbhJIlIbYeGXM6Fqh5Q1XhVraGqNXCu13RS1Tx3hhbC/PmMfIVTm0BE4nGaotYXZJAFxJ9zsRm4HkBE6uMkimgcl3UycI9799PlwAFV3ZbbRiHZ9KSB6/4j7Ph5Ll4BSgL/c6/nb1bVTkELOkD8PBdRwc9zMRNoKyIrgVTgcVWNuFq3n+fi38BbItIX58J290j8YSkin+L8OIh3r8cMBgoDqGoizvWZ9sBa4Chwr1/7jcBzZYwxJh+FatOTMcaYEGGJwhhjjE+WKIwxxvhkicIYY4xPliiMMcb4ZInChBwRSRWRpR5/NXysWyOnnjLP8Jhz3d5Hl7ldXtTNwz56isg97nR3ETnPY9nbItIgn+NcKCLN/NjmUREpfrbHNtHLEoUJRcdUtZnH38YCOm43VW2K09nkK2e6saomquoH7mx34DyPZf9U1ZX5EmVmnGPwL85HAUsUJs8sUZiw4NYcfhCRxe7fldms01BEfnVrIb+JSB23/G6P8nEiEpvL4eYBtd1tr3fHMPjd7es/zi0fJpljgAx3y4aIyGMicjtOn1sfu8cs5tYEEtxaR8aXu1vzGJXHOH/Go0M3ERkrIknijD3xrFv2ME7CmiMic9yytiLys3se/yciJXM5jolylihMKCrm0ew0yS3bCbRR1YuBzsDIbLbrCbyhqs1wvqiT3e4aOgMt3PJUoFsux78J+F1EigLvA51VtTFOTwa9RKQ8cAvQUFWbAC94bqyqE4EknF/+zVT1mMfiL9xt03UGJuQxznY43XSke0pVE4AmQCsRaaKqI3G61L5WVa91u/IYBLR2z2US0C+X45goF5JdeJiod8z9svRUGBjltsmn4vRblNXPwFMiUhX4UlX/FJHrgUuAhW73JsVwkk52PhaRY8BGnG6o6wIbVPUPd/l/gd7AKJyxLt4RkSnAFH9fmKruEpH1bj87fwL1gPnufs8kziI43bZ4nqc7ReQBnM/1uTgD9PyWZdvL3fL57nGK4Jw3Y3JkicKEi77ADqApTk34tEGJVPUTEVkAdACmiUgPnJG8/quqA/04RjfPDgRFpFx2K7l9CzXH6WTudqAPcN0ZvJYJwJ3AamCSqqo439p+xwkswrk+8SZwq4jUBB4DLlXVfSLyPk7Hd1kJMEtVu55BvCbKWdOTCRdlgG3u+AF/x+n8zYuIXACsd5tbvsZpgvkWuF1EKrrrlBP/xxRfA9QQkdru/N+B7902/TKqOg0ngTXNZttDON2eZ2cSzkhjXXGSBmcap9uh3dPA5SJSD2f0tiPAARGpBNyYQyy/AC3SX5OIlBCR7GpnxmSwRGHCxRjgHyKyDKe55kg269wJLBeRpTjjUnzg3mk0CPhGRH4DZuE0y+RKVY/j9K75PxH5HUgDEnG+dKe4+/uR7Nv43wcS0y9mZ9nvPmAVcL6q/uqWnXGc7rWPETi9wi7DGR97NfAJTnNWuvHADBGZo6q7cO7I+tQ9zs8459OYHFnvscYYY3yyGoUxxhifLFEYY4zxyRKFMcYYnyxRGGOM8ckShTHGGJ8sURhjjPHJEoUxxhif/h/DW+pQoXvtxgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fpr,tpr,thresholds = roc_curve(ytest,y_model,pos_label=1)\n",
    "plt.plot(fpr, tpr, color='darkorange', marker= 's', lw=3, label='ROC curve ')\n",
    "plt.plot([0, 1], [0, 1], color='navy', lw=3, linestyle='--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver operating characteristic example')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T16:07:00.422302Z",
     "start_time": "2018-12-19T16:07:00.417421Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.70878837342077916"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "auc(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-19T15:29:49.507861Z",
     "start_time": "2018-12-19T15:29:49.500112Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.70878837342077916"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(ytest, y_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-12T08:09:27.756730Z",
     "start_time": "2018-12-12T08:09:27.083608Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.87055016,  0.86080388,  0.8626922 ,  0.86026436,  0.86458052,\n",
       "        0.86350148,  0.87240356,  0.86377124,  0.8712898 ,  0.86589315])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "cross_val_score(model, X, y, cv=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-12T08:10:14.960968Z",
     "start_time": "2018-12-12T08:10:14.306837Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8655750373019897"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "np.mean(cross_val_score(model, X, y, cv=10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reference\n",
    "\n",
    "1. S. Moro, P. Cortez and P. Rita. A Data-Driven Approach to Predict the Success of Bank Telemarketing. Decision Support Systems, Elsevier, 62:22-31, June 2014\n",
    "2. The dataset and its attribute information are obtained from UCI Machine Learning Repository:\n",
    "https://archive.ics.uci.edu/ml/datasets/Bank+Marketing"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:anaconda]",
   "language": "python",
   "name": "conda-env-anaconda-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
